


I am very pleased to have been invited to come to Szeged to pay homage to my friend, Zoltan 
Bathory. I first met Zoltan when I came to Budapest in 1967 to see Arpad Kiss. When I 
turned up at what was then called OPI, Arpad introduced me to this young, red-haired, thin, 
intense-looking, cigarette-smoking man called Zoltan. Zoltan was clearly some sort of re¬ 
search assistant to Arpad. The reason for my visit was to explain to Arpad and his colleagues 
about the International Association for the Evaluation of Educational Achievement, more 


commonly known as IEA. All ol: the discussion went through an interpreter, then known as 
judit Fiilop. She may remember more about the content ol the discussion than I did. Arpad 
had obviously read about IEA and felt that it might be helpful to Hungary to have some hard 
evidence about the achievement levels of Hungary compared with other countries and also 
something about the major determinants of such achievement. I spent two days explaining 
and answering questions. I was then told that the OPI people had to go and discuss all of this 
with the Ministry people so I was shunted off to Szeged for the day where I met with the 
Faculty of Education staff members. It is of interest that I am speaking English today in 
Szeged. On my first visit, the meeting started in French (mainly I suspect because Agoston 
wanted to show off his French) and it was clear that the faculty members had no idea what 
was being said but sat politely there. I had heard that the older people in Hungary spoke 
German so I switched to German and suddenly everyone woke up!! It also turned out that 
Agoston spoke very good German. But judit still had to translate for a few people but now 
from German to Plungarian. 

At that time I gained the very strong impression that empirical research was viewed with 
great suspicion. Comparing achievement levels was also viewed with a certain fear. In general 
1 sensed a fear to express thoughts openly. How can they explore new ideas openly if they are 
afraid, I wondered. 

In 1968, IEA ran a training course called Seminar on Learning and the Educational Process 
(SOLEP) based on the similar seminars run by Lee Cronbach in Stanford. John B. Carroll 
was the director of the seminar that was held in Skepparholmen near Stockholm. The partici¬ 
pants were new PhDs with a great deal of promise. Ten were from the US and the rest from 
Europe. Arpad brought a lot of pressure that Zoltan be allowed to go. Whereas Zoltan spoke 
fluent German he was still struggling with English. The SOLEP involved interacting in very 
fast English. I watched Zoltan throughout the six weeks. How on earth will he manage, I 
wondered. But he did manage and also learned a great deal. Three years later, IEA ran a 6- 
week seminar on Curriculum Development and Evaluation (led by Benjamin S. Bloom), also 
in Sweden but this time at a place called Granna. In part because of Arpad s pushing and in 
part because of the impression that Zoltan had made at the Skepparholmen seminar, Hun- 
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gary was able to send a team of six Hungarians. This team later became a bit of a mafia for 
curriculum development in Hungary. 

Zoltan soon took over from Arpad as the representative for Hungary in IEA. Zoltan s assis¬ 
tant in Hungary was Peter Vari. Zoltan was by this time interested in research and also in 
curriculum development. But there was always the problem that those with power - the 
major decision-makers in the Ministry — did not listen and were totally unimpressed by “hard 
facts” if they interfered with their previous notions or the ideology that they held. Zoltan 
spent quite a bit of time writing articles in Hungarian that brought in new ideas from other 
countries. 

The “events” came in 1989. A few years after the change Zoltan became a Director-General 
of Education in the Ministry. I think that it was in 1990 or 1991 that I was invited to give a 
presentation to the Education committee in the new Parliament. My presentation was just 
after lunch and I suspect that some members of the committee had indulged a baratsk too 
much. There were seven political parties represented in Parliament and there were thus seven 
members of the education committee. The member from the Farmers’ Party spent most of 
the first half hour soundly asleep. He suddenly awoke and then spluttered “Tell us what they 
do in America, that is what we should do”. So much for careful thought!! 

Zoltan has had the rare experience of being both a researcher furnishing information to the 
Ministry and of being on the receiving end of researchers providing him, but as a policy¬ 
maker now. I have somewhat facetiously sub-titled this article: “the third way”. This is the 
phrase that Tony Blair used to describe his approach to governing the UK. I shall propose a 
new way of having the relationship between researchers and policy-makers and I shall be 
most interested in knowing how Zoltan will react to what I propose. 


Eh ere are several clusters of questions that are typically asked in research about systems of 
education. These include: 

® School buildings: effective least cost architecture, optimum total enrolment, types of labo¬ 
ratories, kinds of sports facilities, maintenance (at what cost and by whom?), classroom 
size and type, etc. 

• Basic equipment for schools and classrooms: Which equipment is required for effective 
learning? 

9 Structure of the system: number of years of primary and secondary education, selective vs. 
comprehensive schools, heterogeneous vs, homogeneous grouping of pupils, length of school 
year, number of terms per year, number of hours of instruction per day and year, curricula 
and syllabi (how detailed?), cross-curricular activities, etc. 

• Teacher training and allocation of teachers to schools: which forms of training are most 
effective for different kinds of outcomes? How are rural and isolated areas best assured of 
good teachers? 

® Homework: How much and how often at different grade levels and how often corrected 
and worked through with pupils? 

9 Which forms of feedback to teachers and correctives by teachers are most effective to 
improve pupil learning? 

• Grade repetition: how much and how determined or should there he none? 

• Decentralisation and cost: how much decentralisation in education should there be and 
how should education be financed? 
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• How is the variance among regions, among schools and among pupils for various forms of 
outcomes is reasonable? How is it checked and how frequently? 

This is but a short list of the many question areas. 'The questions will differ somewhat for 
the different school types: primary, secondary, and vocational school. No education system 
exists in a vacuum. There is a societal and political context as well as a tradition of education. 
Planners of systems must and do take these factors into account. Furthermore there is the 
question of funds available for an education system. If the money is not available, then how¬ 
ever desirable a particular course of action might be there is little use in suggesting it if there 
is no hope of the money being made available. Or even though there is no hope for the money 
now, it might be worth suggesting it now in order to plant the idea in the heads of those in 
charge of the source of funding. Despite these constraints, it can be argued that the planners 
need to know the answers to most of the questions if they arc to be able to take a stance on the 
particular issue. What, then, is the evidence that educational planners and administrators can 
use in order to answer the above questions? 

There is systematic evidence and non-systematic evidence. Systematic evidence is that which 
refers to all schools of interest to the planner (the target population) and also across time (i.e. 
on several occasions). Non-systematic evidence is that which reports an example of a pheno¬ 
menon in a particular school or pupil but that is only one observation (or very few observa¬ 
tions), often a casual one. 


The sources of evidence are historical (through documentary analysis), from current obser¬ 
vation (typically from the inspectorate in a country), and from empirical research conducted 
by educational researchers. However, there are some questions that cannot be answered from 
historical research, lake, for example, the issue of computers in schools. The advent of com¬ 
puters in schools began only in the late 1980s. The field is still developing rapidly and it is 
unlikely that the analysis of historical documents would be likely to have much to say on this 
issue. Each source of evidence is important and not to be discarded lightly. In the best of 
worlds, all three sources should yield the same results. 

It seems to be generally accepted that there is need for “hard” information about the vari¬ 
ous aspects of education. If the policy-makers have such information they are likely to take 
better decisions than without such information. 


The relationship between research and reform — a general picture 

Educational research is a growing industry. Its major aim is to make a contribution to knowl¬ 
edge by supplying the kinds of information needed to answer the questions posed earlier in 
this article. 

In 1899, William James in his Talks to Teachers on Psychology emphasised that education was 
“an art” and not a science and could therefore not deduce schemes and methods of teaching 
for direct application from psychology: “An intermediary inventive mind must make the 
application by using its originality” (p. 8). Nearly 100 years later De Landsheere (1994, 
p. 1871) “Like medicine, education is an art. That is why advances in research do not directly 
produce a science of education, in the positivistic meaning of the term, but yield increasingly 
powerful foundations for practice and decision-making”. Masters, on the other hand, while 
making comparisons with medical research, suggests that in education research syntheses, 
sharper-focussed research questions, and better communications between researchers and 
research utilisers, would do much to improve the link between research and school reform 
(Masters, 1999). 
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In 1960, there seemed to be general agreement in the institution (the National Foundation 
for Educational Research in London) where this author worked that it would take at least one 
generation, about 15 years, between the results being produced by any of the then current 
research projects and their implementation at the school or classroom level. As empirical 
educational research has become a large enterprise, sponsors have increasingly hoped that 
research would provide pointers to the elements of reform that can either help to raise cogni¬ 
tive achievement, affective development and skills or cut the costs of education without dam¬ 
age to the desired outcomes. 


Different aspects of educational research 

Before proceeding with the link between research and reform, it would be desirable to deal 
with the various aspects of research. 

Most policy-makers want, as mentioned above, clear results from research for the problems 
they have. These results should preferably be accompanied by a set of suggestions for reform 
or action. The research from which the results emanate should be “sound”. 

Given that it is often extremely difficult if not impossible to conduct good experiments in 
education from which “cause and effect” can be safely identified, researchers often have to 
depend on natural variation using survey samples. But, this depends on natural variation 
being there for the phenomenon under investigation. Class size is an example of where there 
is some variation in school systems but it is often within fairly narrow limits. Within a coun¬ 
try, the average class size may be 25 children per class at a certain grade level. But the range is 
only 20 to 30. Yet, in other countries, the average class size may well be over 40 or even over 
60. And in these other countries the range will be small. As another example, a country may 
wish to examine the differences between private and public schools but if there are no pri vate 
schools, then there is no natural variation. Furthermore there is always the criticism that 
cross-sectional and longitudinal studies are “correlational” and assumptions must be made 
about “cause and effect”. 

The research must be “sound”. Yet it can be argued that there are many studies that have 
technical deficiencies such that the results cannot be trusted. These deficiencies can include 
poor conceptualisation, poor measurement, poor sampling and incorrect estimates of sam¬ 
pling error, inappropriate analyses, and. false interpretations and conclusions. Some could 
argue that more unsound than sound research is produced in the field of education. The 
International Academy of Education produced a summary of “The requirements of a good 
study” for sample surveys in international educational achievement studies (IAE, Appendix 
2, 1999). This attempts to describe in lay terms the kinds of points in research publications 
that policy-makers should look for when trying to decide whether or not the research was well 
conducted and trustworthy. It would be useful if summaries such as this one were to be 
produced for different types of research. 

It is important that readers of research reports can identify if the research conducted was 
unsound and cannot be trusted. Many senior policy-makers do not have this ability and must 
rely on those skilled in these matters in their own ministries of education. Unfortunately, not 
all ministries have such people. The dangers of implementing policy based on poor research 
are obvious. 

It is also the case that policy-makers need results that have, if possible, been replicated and 
that are generalisable to a grade level or general level (e.g. junior high school or elementary 
levels of education) in an educational system, and where the effects are large. Research that is 
only applicable to a few specific schools (case-study approach) is not of interest to them. 
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Furthermore, the research should ideally be applicable to several key subjects in school and 
not just to one subject. Ministries allocate resources to schools and they want the resources to 
have a wide application in each school and not just to one subject. Unfortunately, having 
results on the effect of variables on several subjects at the same time is costly and time con¬ 


suming. 


Keeves (1994) has distinguished between research studies associated with the generation of 
change and those that serve to maintain and consolidate existing conditions. It is the second 
type of research that he calls “legitimatory research”. Thus the motivation for commissioning 
research must also be taken into account when viewing the links between research and change. 



or policy-making and re scare 


As already stated, policy-makers are interested in their problems within their current frame of 
reference and understanding of education. Many sen ior members of Ministries of Education 
are extremely busy dealing with the day-today problems - and even crises — that arise and, as 
a result, have little time to reflect on the desired goals of education and how best to achieve 
them. Fuller et ah (1990) have very well described the institutional obstacles to framing the 
important policy questions. Many policy-makers think that what researchers generally pro¬ 
duce is “fundamental” research which is of no, or only peripheral, interest to them. What they 
need is research results that they can use directly-applied research. But, what the policy¬ 
makers perceive to be of interest can change with a change of government. For example, the 
matter of busing, educational vouchers, and private schools had quite different priorities 
under the Carter and Reagan administrations in the United States. Equality of educational 
opportunity, on the other hand, has had a high priority under various administrations in 
various countries. 

In the second project of the Southern Africa Consortium for Monitoring Educational Quality 
(SACMEQ), chief policy-makers in fifteen southern African countries were asked to rate 
issues that were of interest to them in primary school ing. One of the issues was that o f private 
tuition outside of school. This was given very low priority despite the fact that the first study 
of SACMEQ had shown the practice to be very widespread and, for the most part, to affect 
achievement even after home background had been accounted for. It was later given a higher 
priority after the researchers had interacted more with the policy makers. 

Policy-makers have allegiances that influence both what they regard as relevant, innocuous 
or even “dangerous” research and also their willingness to take account of research findi ngs. 
Stories abound of ministries or government departments either ignoring research results or 
delaying the publication of government sponsored research because the results did not accord 
with the current political interests. Ways of delaying the publication of research results that 
governments do not like have been hilariously resumed in “Yes Minister” (Lynn and Jay, 
1982). Not only can research be delayed or ignored but only partial results can also be se¬ 
lected to help them argue a case. Keeves (1994) has dealt with many of the problems that arise 
with those in authority when conducting legitimatory research. It is perhaps also of interest to 
note that one of Keeves' first reports was shredded after printing but before publication be¬ 
cause one of the findings negated a policy proposal by one of the State Director-Generals of 
Education. The report contained the results of the first major national analyses of the first 
mathematics study of International Association for The Evaluation if Educational achieve¬ 
ment (IEA). 

Governments, as a rule, last only a few years. The research they sponsor should be com¬ 
pleted within that time period and preferably in less than two years. There are few research 
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projects that can be completed in this time period when it is considered that the instruments 
have to be developed and tried out before the major data collection can begin. The findings 
should be made available in time for the decisions that have to be taken and which will go 
ahead with or without the necessary knowledge base. Operational decision-making cannot 
wait for the results of specifically sponsored research results. 

The researchers often tend to have different backgrounds and value systems from those of 
the policy-makers. Researchers have usually undertaken their work in research institutes or 
university settings. Researchers have been trained to think widely and to question all things. 
Their allegiance tends to be more to fundamental or conclusion-oriented research. They pay 
much more attention to how the quality of their research is received by their research peers 
than by government agencies. 

In most cases, researchers are not involved in the planning and implementation of reform. 
They rarely have any interest in how to plan the costs involved in any reform. They can rarely 
ph rase their resul ts in the language of planning and implementing reform. Indeed, once they 
have published their research report they tend to lose interest in what happens after that. 

Status in the research world depends on the reputation the individual researcher gains from 
his or her research work. Of particular importance is what the research peers say. Status in the 
administrative world tends to be based on seniority and length of service and not on the 
quality of a particular piece of work. 


Models of Research utilisation 

In 1979, Weiss distinguished among seven different models of research utilisation in the 
social sciences: 

• The R&D model. This is a “linear” process from basic research to applied research and 
development to application. Weiss pointed out that the applicability of this model was 
limited in the social sciences because knowledge in the held does not lend itself easily to 
“conversion into replicable technologies, either material or social” (p.427). 

• The problem solving model. Here the results from a specific project are expected to be 
used directly in a decision-making situation. The process consists of: identification of missing 
knowledge — acquisition of research information either by conducting a new study or by 
reviewing the existing body of knowledge —interpretation of the results given the policy 
options available—decision about which policy to pursue. This model has sometimes been 
known as the “philosopher-king” approach where researchers are supposed to provide the 
information from which policy-makers derive guidelines for action. The problem-solving 
model often tacitly assumes about goals but social scientists do not agree among them¬ 
selves about the goals of certain actions. Nor is there necessarily agreement among all 
policy-makers about the goals. 

• The interactive model. This model assumes an ongoing dialogue between researchers and 
policy-makes. This is usually a disorderly set of interconnections and back-and-forthness. 

• The political model. Research findings are used as ammunition to defend a particular 
point of view. It is often the case that policy-makers have already made up their mind 
about taking a particular course of action and should there be research results that legitimise 
this, then they will use it. 

® 1 he tactical model. This is a negative approach in that it is a way of deferring any decision. 
Thus, a controversial issue can be buried or postponed by policy-makers calling for more 
research or further analyses. 
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• The enlightenment mode!. According to Weiss (p.428) this is the model where social 
science research most frequently enters the policy arena. Research can “enlighten” policy¬ 
makers because as research results become available they sensitise informed public opinion 
about ways of thinking of educational problems and come to shape the way in which 
people think about social issues. This is sometimes known as the percolation effect. People 
are helped to redefine problems through research so that they begin to think of the prob¬ 
lems in a different way. 

✓ 

0 Research-oriented model. Weiss refers to this as the “research-as-part-of-the-intellectual- 
enterprise-of-society’ model. Social science research together with other intellectual in¬ 
puts, such as philosophy, history, journalism and so on, contribute to widening the hori¬ 
zon for the debate on certain issues and to reformulating the problems. This is somewhat 
akin the to the enlightenment model outlined above. 

Shavelson (1988, pp. 4-5), when dealing with the utility of social science research, sought 
to reframe the issue of “utility” by suggesting “that the contributions lie not so much in 
immediate and specific applications but rather in constructing, challenging, or changing the 
way policy-makers and practitioners think about problems”. He suggested that one cannot 
expect educational research to lead to practices that make society happy, wise and well-edu¬ 
cated in the same way that the natural sciences lead to a technology that makes society wealthy. 
The assumption that “educational research should have direct and immediate application to 
policy or practice rests on many unrealistic assumptions” (pp. 4-5). Among these are rel¬ 
evance to a particular issue, provision of clear and unambiguous results, research being known 
and understood by policy-makers, and findings implying other choices than those contem¬ 
plated by policy-makers. 

It is, thus, the “enlightenment model” that has been the major way in which research has 
had an effect. 


Some examples 

It is relatively easy to identify some research that has had an enlightenment effect. The work 
of Piaget in identifying “stages of development” began to have an effect on work in curricu¬ 
lum development by the end of the 1950s. Blooms “Taxonomy of Educational Objectives” 
and “Model of School Learning” had a marked effect on many later research projects the 
results of which entered the market place and influenced educational thinking. Colemans 
“Equality of Educational Opportunity” produced the concepts of “school climate” and later 
“social capital ”, both of which entered educational thinking. The “Plowden Report” in En¬ 
gland produced the concepts of parental attitudes and “educational priority zones” both of 
which entered the general educational thinking. Carrolls “model of school learning” and 
later his “human cognitive abilities” research both influenced how educators thought of school 
learning. But, in all of these cases, the process took a long time. In some special cases in 
Sweden in the late 1950s and early 1960s research studies lead directly to some action espe¬ 
cially in the revision of the curriculum. But, the direct effects examples are rare. 

In the first two decades or even more of the work of the International Association for the 
Evaluation of Educational Achievement (IEA) (Postlethwaite, 1994), a massive amount of 
effort went into attempting to identify the major determinants of educational achievement in 
each of the participating countries. It was to the chagrin of the research workers that very little 
attention was given to the results. It was rather the “horse race” national mean achievement 
levels that gained the attention of the Press and the policy-makers. In Italy in the 1970s the 
then Minister of Education talked of the results being like an electric shock because they 
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showed that the gap between north and south Italy was much larger than had been expected. 
The Third International Mathematics and Science Study (TIMSS) of IEA in the 1990s only 
presented aggregate estimates for the test scores and selected questionnaire variables in their 
first set of publications and did not present analyses identifying the major determinants of 
either between country differences, or between-school or between-student differences within 
countries. Nevertheless, it was these aggregated estimates that attracted enormous media pub¬ 
licity and comment. In some cases the comment was appropriate such as about the high 
achievement scores in Japan but the low attitudinal scores towards the learning of the subject. 
In other cases, several of the journalists’ comments and even those of some ministry officials 
proffering policy suggestions assumed that the relationships between certain variables be¬ 
tween countries would be the same as those within countries i.e. they were guilty of commit¬ 
ting the “ecological fallacy” (Ross, 1997). The question must be asked as to why the Press and 
the ministries seem to be more interested in the mean score differences rather than in the 
determinants of such differences. Is it possible that an interface of persons skilled in translat¬ 
ing research findings into policy suggestions is needed? 

The “third way” 

By the beginning of the 1990s, some progress was being made in the attempt to have educa¬ 
tional planners and administrators work more closely with educational researchers in order to 
have more direct links between research and policy-making. One international study in the 
1990s had the strategy of working out in conjunction with ministries of education the re¬ 
search questions at the beginning of the study and then working out the actions to be taken as 
a result of the findings. This latter step was undertaken before the research reports were finalised 
for publication. This study was that of the Southern Africa Consortium for Monitoring Edu¬ 
cational Quality (SACMEQ). The last chapter of each national publication listed the sug¬ 
gested policy actions categorised by cost and time needed for implementation. An example is 
given in Table 1 of selected suggested policy actions for the Mauritius report (Kulpoo, 1998, 
pp. 86-92). There were 38 policy suggestions in all. (Table 1) 

It should be noted that the units undertaking the research study were units within the 
ministry of education and not special research units either within the ministry or in higher 
education institutes. The actual suggestions were checked, and in one or two cases modified, 
by the senior Ministry personnel before the book was published. To what extent the sugges¬ 
tions were implemented is not yet known. On the other hand, there was a similar study in 
Zimbabwe with similar policy suggestions in 1991 and a report documenting the implemen¬ 
tations of the suggestions has been published (Ross, 1995). Despite the fact that certain of the 
suggestion were implemented there was no improvement in Grade 6 reading achievement by 
I 997 (Shumba, 1998). It is unclear whether the implementation was not well conducted or 
whether only some of the easier but not crucial suggestions were implemented. This will also 
apply to the other SACMEQ reports. 

The very large sample survey of Grade 5 achievement and its determinants that is under¬ 
way in Vietnam has also taken this approach. The Ministry first decided on the policy ques¬ 
tions that it wanted. Some of the many policy questions that it had have been presented in 
fable 2. it can be seen from the table that reference is made to the dummy tables and ques¬ 
tionnaires referring to the particular policy questions. The data analyses will only be con¬ 
ducted in late 2001 (some six months after this speech) but should result in a series of policy 
suggestions in the same style as the Mauritius example given earlier. 
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Table 1: Selected Policy-Suggestions from the S'ACMEQ_ study in Mauritius 



1. The Ministry shouid ask the Staffing Branch to meet with Teacher Union 
representatives to discuss the potential “conflict of interest” that arises from the 
current practice of allowing the teachers of around 80 percent of Grade 6 pupils 
to receive high financial rewards for teaching their own pupils as Private clients 
outside of school hours. 


Staffing 
Branch and 
Teacher Union 


Short 
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2. The Inspectorate should be asked to review and, if necessary, establish 
benchmark standards for the educational environment that are deemed to be 
“reasonable for the proper functioning of primary schools”. 

| Inspectorate 

Short 

L___--- 

Low 

Data collection for planning purposes 

3. The Supply Branch of the Ministry should investigate the reasons for the 
relatively high percentage of Grade 6 pupils in Rodrigues who reported that they 
lacked basic learning materials. 

Supply Branch 

Medium 

Low 

Education policy research projects 



4. The Planning Unit should undertake an investigation into the practice of grade 
repeating in Mauritius in order to determine whether the extra year of schooling 
being received by the “average Grade 6 pupil” can be justified on either 
educational or economic grounds. 

5. The Ministry should plan to undertake a follow-up survey of the same target 
population employed during SACMEQ’s initial project in order to examine changes 
in important educational indicators over time. 


Planning Unit Medium Low 


Planning Unit Medium Moderate 


Pre-requisites for the “third way ” to work well 

The key element in this process is the first step of the Ministry and researchers establishing 
the policy questions. The researchers have to be able to put themselves into the way of think¬ 
ing of the Ministry people. The Ministry people have to be able to stand back from their daily 
tasks and think past todays crisis to what the longer term goals for the education system are. 
This is not an easy role change for either side. In my experience it is best to have the heads of 
divisions in the Ministry for a good half day The minister shouid be able to arrange this and 
chair the meeting. It is also desirable if the researchers listen carefully and then draft what 
they think the meeting was suggesting for the policy questions. The notion of integrating 
research and training is also important. Assuming that the researchers are part of the Ministry 
this integration lays the basis for this in Ministries beginning to think about “informed solu¬ 
tions to policy issues 55 rather than “inspired guesswork”. This approach provides a more genu¬ 
ine Ministry “ownership 55 . Where the research is international it is clear that the researchers 
involved can learn from each other. They learned by doing not only in terms of new tech- 
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niques but also in terms of ways of operationalising questions about educational issues. Those 
international projects that “dump” centrally designed studies on countries where the role of 
the researchers in the countries has been reduced to that of glorified field workers are to be 
deplored. 

The demand by Ministers of Education to be involved in SACMEQwas high. In a rela¬ 
tively short time there were 15 Ministries of Education from 15 countries that were members 
of SACMEQ. But governments change and Ministers change. The new Minister did not 
necessarily have the same learning experiences as his predecessor. In two of the SACMEQ 
countries the new Ministers did not allow their researchers to participate in Phase 2 of the 
study even though there were several senior members of the Ministries who said that they 
needed the information that the study would have yielded. On the other hand there were 
other countries in different parts of the world where the Ministers were saying that they 
should be involved in such studies. In general this model of doing research is very popular 
with Ministries of Education. 

In some cases the implication of the question is spelled out but in other cases this has not 
been done because it is felt to be self-evident. 


It should be noted that the governing body of SACMEQ consists of the 15 Ministers of 
Education-. It is the governing board that decides on the kinds of research projects that 
SACMEQ will undertake. The ministries participating in the study set the research ques¬ 
tions. The instruments and data belong to SACMEQ. The International Institute provided 
certain technical expertise for Educational Planning (IIEP), a UNESCO institution. 

The “third way 5 has advantages and disadvantages. T he advantages are: 

• It is the Ministry s research questions in which they have an immediate and direct interest 
that are posed 

@ Over time the senior members of the Ministries become accustomed to reflecting on the 
educational issues in their countries and setting the research questions 

• Over time the researchers become used to having to undertake research strictly based on 


Ministry questions 

@ The Ministry ensures that all selected schools actually participate 
® Funds are made available to carry out the study 

® The Ministry is waiting for the results and, where possible, the members of the ministries 
help in the interpretation of the data 
s The Ministry, where possible, acts on the policy-suggestions made 
s This form of research becomes to be taken as a continuing exercise 
The disadvantages are: 

s It is only Ministry questions that are asked. There may be other very important questions 
that could be asked but that were not posed by the Ministry personnel. This raises the 
question of whether other groups should also be involved in setting the research questions. 
If so, then which groups? 

8 It is typically the Ministry’s planning unit that is involved in the conduct of the research. 
Quite often these planners need training in research techniques. Once trained and they 
have acquired certain skills they leave and go to the private sector. This requires that Min¬ 


istries establish career possibilities such that skilled persons do not leave. 


Ministries are capable of suppressing publication of results if the results are embarrassing 
for the Ministry 
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Table 2: Policy questions, tables and questionnaires for Vietnam- Grade 5 study 


Tabulation of policy questions, tables in chapters addressing them, and references to questions in the 
questionnaires* 


Group 

Question 

number 

Policy question 

1 Table 
reference 

j Questionnaire 
reference 



Pupils 



01 


What are the characteristics, including home background, of the 
Grade 5 pupils? What actions do these characteristics require the 
Ministry to take? Do these characteristics and the home back¬ 
ground have an influence on achievement? 

1 -— 

■ 



1.01 

What is the age distribution of Grade 5 pupils? Are there distribution 
patterns requiring corrective action and/or having an influence on 
teaching methods and/or the curriculum? 

3.1 

P02 D 


1.02 

What is the sex distribution of Grade 5 pupils? Are there imbalances in 
the enrolment of male and female pupils requiring corrective action? 

3.1 

P03 D 


1.03 

What is the ethnic group of the children? 

3.1 

P05 D 


1.04 

How regularly do Grade 5 pupils eat meals? 

3.1 

P10 


1.05 

What percentage of Grade 5 pupils speaks the language of the test at 
home? 

3.2 

P04 D 


1.06 

What is the level of the parents’ education of Grade 5 pupils? (Add) 

3.1 

Pll D, P12 D 

.... 

1.07 

How many books are there in pupils' homes? 

3.1 

P06 D 


1.08 

What is the socio-economic status of the pupils and their parents? 
(Measured in terms of possessions, parental education, and books at 
home) 

3.1 

P07 D and R, 
P06, 

Pll + P12, 

R for ses 


1.09 

Do the pupils have a quiet corner at home in which to work? 

3.2 

P08 


1.10 

How many hours per day did the pupils have to work to help their 
families? 

3.2 

P09 D 


1.11 

How many minutes do pupils have to travel to school? 

3.2 

P13 


1.12 

On how many days were the pupils absent from school in the previous 
month? 

3.2 

P14 & 15 


1.13 

What percentage of pupils had repeated at least one grade? 

3.2 

P16 D 

01.1 


What was the situation about homework? 




1.14 

To what extent did pupils get in their home regarding homework and 
interest in their school work? To what extent was the homework 
corrected by teachers? 

3.3 

In text 

P24 D, P25 D, 
P26, P30, 

P33, 

P31 D, P34 D 

01.2 


How many pupils had extra tuition, was it paid for and who taught it? 




1.15 

How many pupils had extra tuition, was it paid for and who taught it? 

\ . ... 

3.4 

P27 D, P28 

D?, P29 


* P = Pupil questionnaire D = Derived variable to be computed (see Vietnam: recoding of variables) 

TQ = Teacher questionnaire R = Rasch score to be computed S = School Head questionnaire 
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Group 

Question 

number 

Policy question 

Table 

reference 

Questionnaire 

reference 

05 


How did the schools’ conditions match the Ministry’s 
benchmarks? 

M0ET to produce benchmark 
information to be used 


5.01 

How did the schools meet the Ministry’s benchmarks (and other 
benchmarks) for total enrolment, class size, classroom space, total 
space and staffing ratio? 

7.1 

7.2 

Use derived 
variables 


5.02 

How did the schools meet the Ministry’s benchmarks (and other 
benchmarks) for sitting places, writing places and a board? 

7.3 

7.4 

TQ15.1, TQ15.2, 
TQ17.2 


5.03 

How did the schools meet the Ministry's benchmarks (and other 
benchmarks) for classroom supplies? 

7.5 7.6 

P21.01, P21.02, 
P21.03, P21.04, 
P21.05, P20.01, 
P20.03, P20.06, 
P20.10 


5.04 

How did the schools meet the Ministry’s benchmarks for the aca¬ 
demic and professional qualifications of both teachers and heads? 

7.7 

S25, S 26, 

S8 S9 

06 


To what extent were material and human resources allocated 
equitably among provinces and among schools within provinces? 




6.01 

What was the resource distribution variation among schools within 
provinces and among provinces for classroom furniture, classroom 
supplies, a classroom bookcase, classroom space per pupil, and 
school resources? 

8.1 

Use derived 
indexes for the 

measures. 

;l 

6.02 

What was the resource distribution variation among schools within 
provinces and among provinces for teacher and school head 
qualifications (academic and professional), years of teaching 
experience, experience as a school head, inspectors’ visits and the 
pupil-teacher ratio? 

8.2 

S25, S26, S8, S9, 
S34, TQ7.1+7.2 
Derived variable 
for P-T ratio 

07 


What was the achievement in reading and mathematics {and 
various sub-scores in these two subjects for provinces, the 
country as a whole and various sub-groups of pupils? 




7.01 

What were the scores in the various domains of reading and total 
reading for each province and for Vietnam? 

9.5 

Narrative score R 
Expository score- R 
Documents score- R 
Total score R 

Province ID 


7.02 

What were the scores in the various domains of mathematics and 
total mathematics for each province and for Vietnam? 

9.6 

Number R 
Measurement R 

Space and data-R 
Total R 

Province ID 


7.03 

What were the scores in reading for the essential items, minimum, 
and desirable levels? 

9.7' 

Essential Read 
Minimum Read 
Desirable Read 
Province ID 

i 

7.04 

What were the scores in mathematics for the essential items, 
minimum, and desirable levels? 

9.8 j 

Essentia! Math 
Minimum Math 
Desirable Math 
Province ID 
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My overall impression is that whichever form of relationship exists between the users and 
producers of educational research, it can be very fragile. Much depends on the prior knowl¬ 
edge of the senior members of Ministries of Education. But this “third way” would appear to 
work well in many countries. Would it work well in Hungary? Does the Hungarian Ministry 
have the pre-requisites that I have outlined above? Is it open to other ways of looking at 
problems? How many secondary analyses of existing data sets has it undertaken itself? Does it 
encourage participation in international research endeavours? Does it hold regular seminars 
for its staff members on research findings and their implications for the Hungarian system of 
education? And so on. Equally how do the researchers in OKI and elsewhere in the country 
operationalise their policy suggestions based on their research in ways similar to those exem¬ 
plified in Table 1 earlier in this speech? 


X Neville Postlethwaite 
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